Scripting distributed scientific workflows using Weaver

ion Description Run(function, inputs) Apply inputs to function. Map(function, inputs) For each input in inputs, apply function. MapReduce(mapper, reducer, inputs) For each input in inputs apply mapper, sort intermediate outputs, and then apply reducer. AllPairs(function, inputs_a, inputs_b) Apply function to all combinations of inputs_a and inputs_b. Wavefront(function, matrix) Compute recurrence relation by applying function to matrix in a wave pattern. Copyright © 2011 John Wiley & Sons, Ltd. Concurrency Computat.: Pract. Exper. 2012; 24:1685–1707 DOI: 10.1002/cpe SCRIPTING DISTRIBUTED SCIENTIFIC WORKFLOWS USING WEAVER 1691 inputs to generate the outputs. Although very simple, this abstraction is useful for constructing more complicated patterns of execution. 2.3.2. Map. The Map abstraction is a common pattern used for work that exhibits data parallelism. Map in an input function, which is applied to each item in the input dataset. The results of each function application is stored in a collection of output data objects or as a single data file if the user specifies an output target. Because each function application is independent of other function executions, the individual tasks in this pattern are data parallel and thus can be executed concurrently. 2.3.3. MapReduce. Another common data-processing abstraction provided by Weaver is MapReduce [1]. In this pattern, a mapper function is applied to the initial set of inputs to generate a group of intermediate output files that are partitioned, sorted, and then passed to the reducer function for aggregation. All the tasks in both the mapper and reducer phases exhibit data independence and therefore can be run in parallel. 2.3.4. All-Pairs. An abstraction that is frequently used in fields such as biometrics and data-mining is All-Pairs [2]. In this pattern of work, each member of one dataset is compared with each member of another dataset to produce a matrix that contains the scores for each comparison. Like the previous abstractions, the individual comparison tasks can execute independently of each other, which allows the jobs to be scheduled to run concurrently. 2.3.5. Wavefront. An abstraction used in game theory and gene sequencing applications is Wavefront [13], which computes a two-dimensional recurrence relationship where each cell in the output matrix is generated by a function whose arguments are the values in the cell immediately to the left, below, and diagonally left and below. Although some cells can be processed in parallel, due to the recurrence relationship, special care must be taken to ensure the proper ordering of dependent cell computations. By default, Weaver includes all five of these abstractions as a part of the framework. However, because abstractions are just normal Python functions, it is possible for users to extend the existing ones or define their own abstractions specific to their workflow.